Skip to content

Add max_length, convert more of each page and make extraction faster - #23

Merged
jasir-serpapi merged 77 commits into
mainfrom
jasir/max-length-and-fixes
Nov 12, 2026
Merged

jasir-serpapi merged 77 commits into
mainfrom
jasir/max-length-and-fixes

Conversation

@jasir-serpapi

@jasir-serpapi jasir-serpapi commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • New option: max_length: returns exactly the start of the full Markdown and stops rendering early on large pages.
  • main_content: false: now converts everything visible, so a selection of a page comes out whole.
  • More content kept: native players, link targets kept in data-* attributes, lazy-loaded images, image-only links, heading-only pages, single-column tables, and the headline above an article body.
  • Safer Markdown: text, URLs and link labels can no longer form links, autolinks, tags or code spans that the page didn't contain.
  • Faster: Wikipedia's Coffee article goes from 364 to 214 ms in main mode and 109 to 92 ms in full mode; the speed commits leave output unchanged.
  • Robustness: markup nested more than 64 levels deep no longer raises SystemStackError.

max_length

  • nil, the default, changes nothing.
  • An Integer n returns exactly to_markdown(html)[0, n] and the same excerpt.
  • A negative or non-Integer value raises ArgumentError.

The renderer stops after about max(2n, 1000) characters, at a block boundary. Cleanup keeps only blocks that the full cleanup would also keep at the start. Some later content can still change the start: an image whose copies so far are all redundant, an H1 further down that could replace the title, or a missing excerpt paragraph. In those cases the whole content is rendered once. A page therefore costs at most one short render plus one full render.

Large page (tmp/bench/big.html, 20,000 paragraphs), nil vs 50_000:

Mode nil 50_000
Main 959 ms 838 ms
Full 141 ms 30 ms

Main mode gains less because content selection, not rendering, dominates there.

Behaviour changes

main_content: false

  • Kept now:
    • headings without a section;
    • boilerplate lines such as "Advertisement" and "Share";
    • repeated headings, paragraphs and images;
    • quotes and rules;
    • button and form text;
    • open dialogs.
  • Still dropped:
    • scripts and styles;
    • hidden elements;
    • form inputs;
    • closed dialogs;
    • punctuation-only lines.
  • On WCXB, 1707 of 2008 full-mode pages change, with 3.3% more characters.

Links and media

  • <video> and <audio> become [title](file). Muted background videos are skipped. A player inside a link is replaced by its thumbnail, unless the player is hidden.
  • An iframe embed is linked only when its src is an http(s) URL on the video host's own domain. An embed without a title is written as [Video](url).
  • A link with no href, or a blank one, takes its target from data-href, data-url and similar attributes.
  • A link made only of images keeps its target: [![alt](src)](href).

Images

  • Lazy-loading attributes win over a placeholder, spacer or thumbnail src.
  • They are read in priority order: high-resolution names first, then the largest size in data-a-dynamic-image, then the other loaders' names (thumbnails last) and the largest image of a lazy srcset.
  • A value that is not an image location never replaces an image's src, whichever attribute, srcset or size map holds it: a word, false, # or a number. An image without a src still takes a bare name such as portrait, but not false, null and the like.

Content selection

  • When an article body is chosen inside <main>, the H1 above it now opens the content. The deck or subtitle is left out on purpose: keeping it lowered ScrapingHub and Mozilla F1.
  • A page made only of headings keeps all of them, not just its H1.
  • A one-column table with a <th> or a caption stays a table.
  • A table cell escapes every pipe, so a backslash before a pipe stays. A table nested in a cell is escaped once, for the outer table.

Markdown safety

  • Disguised javascript: and vbscript: links are dropped, for example java&#9;script:.
  • Brackets and backticks in URLs are percent-encoded, and URL parentheses are kept only where Markdown pairs them.
  • Link text is closed at its own bracket. Links, autolinks, tags and code spans inside it are escaped.
  • Raw TeX, emoji alt text, email autolinks, a trailing backslash, long backtick runs, and text pieces placed side by side can no longer start Markdown that runs past them.

Performance

Ruby 2.7.8, median of 10 runs (100 for books), origin/main → this branch:

Page Main Full
Wikipedia Coffee 364.9 → 213.9 ms 109.0 → 91.5 ms
books.toscrape 2.8 → 2.1 ms 1.0 → 0.7 ms
tmp/bench/big.html (5.4 MB) 1269 → 959 ms 284 → 141 ms

WCXB benchmark, in ms per page: dev 30.2 → 22.0, test 23.1 → 16.6. Over all 2008 WCXB pages, p50/p99: main 18.7/166.9 → 13.3/111.3 ms, full 9.9/62.3 → 8.3/54.2 ms.

Where the speed comes from:

  • each furniture class name is matched once;
  • each class signature is computed once per extraction;
  • named ancestors are found by walking up the tree;
  • plain-ASCII text takes fast paths;
  • each URL is resolved once per page;
  • paragraph divs are found from the blocks inside them;
  • class names are checked for a furniture word before the slow patterns run;
  • trees are walked through element pointers instead of Lexbor selectors and Node#traverse.

Each of these changes was checked to give identical output on all 2008 WCXB pages, in both modes.

Quality

F1, origin/main → this branch:

Corpus F1
ScrapingHub 0.947 → 0.947
Mozilla 0.948 → 0.949
WCXB dev 0.876 → 0.875
WCXB test 0.900 → 0.900
  • Mozilla gain: embed URLs are no longer scored as words.
  • WCXB dev loss: about 0.0001, from two documented cases:
    • The linked-image change produces an alt text, [X-Info], that the benchmark's Markdown stripper can't parse.
    • On some pages the H1 replaces a near-duplicate metadata title.

Testing

  • bundle exec rspec: 412 examples, 0 failures, 1 pending.
  • The pending example checks the max_length prefix on every WCXB page. Run it with WCXB=tmp/corpora/wcxb bundle exec rspec. It passes in about 5.5 minutes, checking lengths 0, 1,000, 5,000 and 50,000 in both modes.
  • Each fix comes with a spec that fails without it. Speed changes that depend on a pattern property have a spec guarding that property.
  • Each commit message gives its WCXB, ScrapingHub and Mozilla results.

Notes for reviewers

  • VERSION is not bumped.
  • CHANGELOG.md → Unreleased covers every change above.

jasir-serpapi and others added 30 commits November 9, 2026 18:14
Page analysis and rendering walk the DOM recursively, so a few hundred
levels of nesting exhausted the stack of a thread, which is much smaller
than the main thread's; Puma serves requests in threads. 640 nested divs
raised SystemStackError in Stats, 300 nested spans in the renderer, and
4000 divs or 10,000 divs with attributes raised even on the main thread.
Renderer::MAX_DEPTH bounded only block rendering, not inline rendering
or the walks before it.

The preprocessor now replaces the children of every element 64 levels
below <body> with their text, collected without recursion, after
<noscript> content is unwrapped and hidden elements are dropped. Block
boundaries and <br> become line breaks so words stay apart. Tables and
definition lists get a cell or <dd> around the text, which their
renderers would otherwise skip. Real pages nest about 20 levels deep: on
2319 pages of WCXB, Scrapinghub and Mozilla the deepest is 54, and no
output changes. With nesting bounded, MAX_DEPTH is gone.

At 3000 levels every shape renders inside a thread with about 9,500
spare Ruby frames on Ruby 2.7, 3.4 and 4.0.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The preprocessor removed <video>, <audio> and <source> as junk, so a
page's own clips and podcast episodes vanished from the Markdown, as on
W3Schools' HTML video page.

A player now becomes a paragraph of its own, [title](file), using its
src or its first <source>. Brackets and backslashes in the title are
escaped; without a title the label is "Video" or "Audio". The fallback
text inside the player is dropped. Muted videos are autoplaying
backgrounds and a player inside a link previews the linked page, so
both are skipped, as are blob: and data: sources, which only the
browser that made them can play. Conditional cleaning keeps a container
with a player, as it keeps one with a video embed.

On WCXB, 13 pages gain 34 media links in main-content mode and 23 pages
in full mode; F1 is unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Some sites build links with scripts and keep the target in data-href or
a similar attribute instead of href. Those anchors rendered as plain
text.

An anchor without href now takes the first non-blank value of data-href,
data-url, data-target-url, data-delayed-url, data-fallback-url, ourl or
rl-link-href. A template placeholder such as "mailto:?subject={title}"
is no target and is skipped; javascript: and other non-web targets stay
text as with href.

On WCXB, 3 pages change in main-content mode (forum post permalinks and
product page promos) and 4 in full mode.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An image's data-src was used only when src was missing, a data: URI or
the class said "lazy". Many loaders mark nothing but the attribute, so
an image with a spacer file as src was dropped as decorative, and one
with a low-quality thumbnail as src rendered the thumbnail.

The attributes only loaders use (data-src, data-lazy-src, data-original
and the like) now win over any src when they hold a path or a web URL,
not a word such as "false" or a caption. Generic names such as data-url
and data-image, which often hold something else, still count only on a
marked image. A src that matches the tracking-pixel and placeholder
pattern (spacer.gif, blank.gif, placeholder.png) counts as missing, as a
data: placeholder already did.

On WCXB, 10 pages change in main-content mode: placeholders and eBay's
1x2 spacer become product photos, Times of India's generic thumbnail
becomes each story's image, and WooCommerce thumbnails become the full
images. 22 pages change in full mode. F1 is unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Markdown cleanup dropped headings without a section and short lines
such as "Advertisement" in every mode. With main_content: false that
guesses at what is content, which the caller turned off. Converting
only the h2 elements of Wikipedia's Coffee article this way gave empty
Markdown, and its h1, h2 and h3 gave only "# Coffee".

In full mode, cleanup now keeps headings without a section and lines
that match the boilerplate patterns or hold only symbols such as rating
stars. It still drops lines of bare punctuation ("•", "..."), repeated
blocks, stray rules and repeated images. Main-content extraction is
unchanged.

On WCXB, main-content output is unchanged. Full-mode output keeps 1856
more blocks on 922 pages, mostly "Skip to content", "Share" and
"Advertisement" lines. On the Coffee article, the h2 elements give the
16 section headings (259 characters) and h1, h2 and h3 give 45
headings.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate matched every element's whole class signature against the
union of the chrome and noise patterns. On Wikipedia's Coffee article
almost every signature is unique (5,911 of them), so the signature
cache never hit and the union regex alone took about 47 ms per page.

Class names repeat far more than class lists do, and no furniture
pattern can match across the space between two names, so the signature
is now furniture when one of its names is, with a cache per name. A spec
guards the pattern property this relies on. On all 2,008 WCXB pages no
output changes, in either mode.

Wikipedia Coffee, main content, Ruby 2.7.8: Boilerplate 92 -> 68 ms,
whole extraction 378 -> 359 ms (median of 10).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate, the content locator, the scorer and the pruner each read
the signature of the same elements again, and building one (camelCase
split, generated-id stripping) took 8% of a Wikipedia extraction:
27,718 calls for 5,372 distinct class and id strings.

Extractor#call now keeps a cache of signatures by their raw class and id
string for the length of one extraction. It is fiber-local, so threads
share nothing and nothing outlives the call; code that runs outside an
extraction computes signatures as before. The camelCase split uses a
lookaround instead of capture groups, with the same result. On all
2,008 WCXB pages no output changes, in either mode.

Wikipedia Coffee, main content, Ruby 2.7.8: 359 -> 344 ms (median of
10). WCXB main mode p50 19.4 -> 18.8 ms, p99 167 -> 159 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Dom.ancestors_named asked Nokolexbor for every ancestor, which builds a
NodeSet of Ruby objects each time, and then filtered it. The pruner
calls it for every paragraph, div and span in the article to skip lines
inside tables, so it took 3.6% of a Wikipedia extraction. Walking up
through #parent and keeping the matching elements gives the same list.

On all 2,008 WCXB pages no output changes, in either mode.

Wikipedia Coffee, main content, Ruby 2.7.8: Pruner 107 -> 96 ms, whole
extraction 344 -> 334 ms (median of 10). WCXB main mode p50 18.8 ->
18.2 ms, p99 159 -> 149 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Text.collapse, Text.escape and Dom.squish run on every text node and
element text, and together took about 20% of a whole-page conversion of
Wikipedia's Coffee article. Most of that text is ASCII and needs no
escaping: of 7,698 text nodes, 6,923 are ASCII and 297 hold a character
escape could change.

ASCII text now goes through a five-character tr table instead of the
Unicode spaces table, invisible characters are deleted only from
non-ASCII text (they are all non-ASCII), and escape returns text with no
escapable character untouched instead of running six gsubs over it. On
all 2.7 million text nodes of WCXB the new and old versions agree, and
on all 2,008 pages no output changes, in either mode.

Wikipedia Coffee, Ruby 2.7.8, median of 10: main content 334 -> 314 ms,
whole page 107 -> 92 ms. WCXB main mode p50 18.2 -> 16.8 ms, p99 149 ->
137 ms; full mode p50 10.2 -> 8.6 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Content selection (RedundantImages) and rendering both resolve every
link, and a page links to the same places many times: Wikipedia's
Coffee article has 2,164 links to 1,813 targets, each parsed, merged
and cleaned through URI, about 6% of the extraction. UrlResolver, which
lives for one page, now remembers the result for each href, image src
and same-page check. Results are frozen so no caller can change a
cached value.

On all 2,008 WCXB pages no output changes, in either mode.

Wikipedia Coffee, main content, Ruby 2.7.8: render 44 -> 32 ms, whole
extraction 314 -> 299 ms (median of 10). WCXB main mode p50 16.8 ->
16.6 ms, p99 137 -> 132 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Dom.paragraph_keys visited every node below the root and asked each
element for its children, to find the divs with no block inside them.
The scorer and conditional cleaning both run it, and it took 8% of a
Wikipedia extraction. It now selects the elements that break a
paragraph (div, p, img, lists, tables...) with one CSS query and marks
their ancestors, then keeps the p elements and the unmarked divs.

The walk up does not stop early at an ancestor already marked: Lexbor
splits a link cut by a block into copies that share one attribute list,
and so one key, which stopping early mistook for being done. The old
traversal let such a key's last copy decide; now any copy holding a
block marks the key, which is also what the spec for misnested links
expects. On WCXB the old and new results differ for one page body
(4190, a minicart link split around a footer), and on all 2,008 pages
no output changes, in either mode.

Wikipedia Coffee, main content, Ruby 2.7.8: content locator 56 -> 42 ms,
pruner 88 -> 72 ms, whole extraction 290 -> 263 ms (median of 10). WCXB
main mode p50 16.6 -> 15.3 ms, p99 132 -> 126 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The check for javascript:, vbscript: and data: links ran on the raw
attribute, before URL resolution deleted the tabs and line breaks a
browser ignores. So href="java&#9;script:alert(1)" became
[x](javascript:alert(1)), and the data-href targets and native players
added on this branch opened two more ways in. Leading control
characters got past it the same way, and a <base> with a script scheme
was only refused when written in lowercase with no leading space.

UrlResolver now normalizes a URL as a browser reads it and refuses
script schemes before and after resolving, so neither an href nor a
<base> can produce one. Pruner's link check uses the same rules.

Markdown decodes backslash escapes and entity references in a link
destination, so "javascript\:" and, without a page URL to resolve
against, "javascript&colon;" still read as script links. Backslashes in
a destination are now percent-encoded, and so is an entity reference
before the path of a URL that has no scheme.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A native player's title only had its brackets and backslashes escaped,
so a title holding "<img src=x onerror=alert(1)>" (entity-encoded and
inert in the page) became raw HTML in the Markdown, and asterisks or
underscores in it became emphasis.

The title now goes through the escaping all text gets, and then every
bracket in it is escaped, so no title can end the label. A backslash
the text escaping already wrote keeps its character; a lone one at the
end is doubled so it cannot escape the closing bracket.

On WCXB, the four players with a title have nothing in it to escape,
so no output changes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Each player searched all its descendants for a <source> and walked up
to <body> looking for a link, before deep elements were flattened. In
markup that nests players, or puts one at every level of a deep nest,
that is quadratic: 20,000 nested <video> took 3 s and 10,000 levels of
divs each holding a player 11 s.

A player now uses its src when it has one, and otherwise only a
<source> that is its own child, which is also where HTML reads sources
from, so a nested player's file is no longer taken for its parent's.
The walk up stops at MAX_DEPTH levels: a player deeper than that turns
into plain text anyway. The three shapes now take 0.1 to 0.25 s.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A layout table cell with no text and no image was dropped as a spacer,
so a <video> or <audio> that was the only thing in its cell vanished:
<table><tr><td><video src="/clip.mp4"></video></td></tr></table> gave
an empty page. The same check counted such cells as blank when telling
data tables from layout ones.

A cell holding a player is now content in both places, like one
holding an image.

On WCXB, output is unchanged in both modes: no page puts a player alone
in a table cell.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Conditional cleaning deletes a container whose class weighs negative
before it looks at what the container holds, and "media" is one of
Readability's negative words. So <div class="media"> holding a <video>
and a caption paragraph lost both, though a container holding only a
player was kept.

On a container that holds a player, "media" in its class now names the
player and weighs nothing. The other negative words still count, so a
"sidebar-media" player is still removed with its sidebar.

On WCXB, one page changes in main mode: Ring's doorbell page (4261)
keeps the product video in a "media-position__bottom" tab. Citizen's
Bulova page keeps its "hero-media" background videos as before. Full
mode is unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
An image's lazy-loading attributes were checked unevenly. Without a
lazy class, a data-src naming a spacer file still replaced a real src,
so <img src="/real.jpg" data-src="/spacer.gif"> was then dropped as a
spacer. On an image whose src was a placeholder, or with a lazy class,
the first non-empty attribute won whatever it held: data-src="false"
became an image at /false even when data-original held the photo, and
data-src="#" an image at the page's own URL.

Every lazy-loading attribute now has to hold an image location that is
not a spacer or placeholder file, and a bare fragment is not a
location. The first that does wins; otherwise src stays.

On WCXB, one page changes in full mode: Leadfeeder's hero image, whose
data-src is "#", now comes from its <picture> source instead of being
a link to the page itself. Main mode is unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Converting the whole page keeps a line of rating stars, but not when
the stars shared a block with an image repeated elsewhere: once the
repeat was removed, the block was kept only if it held a letter or a
digit, so <p><img src="/a.jpg"><br>★★★</p> after the same image lost
its stars too.

In full mode such a block is now kept by the same rule as any other,
a letter, digit or symbol, and stray punctuation is still dropped.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Finding paragraph divs from the blocks inside them changed which divs
count as paragraphs when the parser had copied a link or a <b> cut by a
block. The copies share one attribute list, and so one key, and any
copy holding a block now marked the key, where the bottom-up pass let
the copy visited last decide. Content selection and cleaning were tuned
with the old answer, and on such pages the new one lost content: a
link split around a <section> took both of its lines with it, and a
page whose unclosed links wrapped the article dropped the whole tail.

A root whose formatting elements share a key now keeps the bottom-up
pass. Others take the fast path, which can now also stop at an ancestor
already marked, since no key there stands for two elements. Finding
shared keys with XPath costs about 1 ms on Wikipedia's Coffee article.

The paragraphs found match the bottom-up pass on all 43,516 WCXB roots
checked and on 20,000 random malformed trees. 59 of 2,008 WCXB pages
have copied links; output is unchanged on all of them, in both modes.
Wikipedia Coffee, main content: 265 ms, as before (median of 10).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Normalizing a URL removed the spaces and control characters around it
with one regex, and on Ruby 2.7 its trailing alternative rescans a run
of spaces inside the URL from each of its characters. An href holding
40,000 spaces between two letters took over 2 s, 50,000 took 9 s.
Ruby 3.4 and 4.0 are not affected.

The ends are now found with index and rindex and the URL sliced
between them, which gives the same result on 200,000 random strings of
spaces, controls and letters. The 50,000-space URL takes milliseconds.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A link destination kept its parentheses whenever it held as many "("
as ")". In "mailto:x)[Run](javascript:alert(1))(" the first ")" ends
the destination, and the rest of the URL is read as a second link, to
javascript:alert(1). Vertical tabs and other control characters were
also written raw, and Markdown ends a destination at them.

Parentheses now stay only when every ")" closes an earlier "(" and they
nest at most 32 deep, as far as CommonMark reads them; otherwise all of
them are percent-encoded. Control characters, spaces and DEL are
percent-encoded too. CommonMarker reads each repro as a single link
with no script href, and Wikipedia's Coffee_(disambiguation) link is
unchanged.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Skipping words and spacer files among lazy-loading attributes applied
the rule meant for unmarked images to every image. On an image with a
lazy class or a placeholder src, a data-src of "logo.svg" or
"/my photo.jpg" was rejected for not naming a raster file or for
holding a space, and the image was dropped; main kept both.

A marked image now takes any path, web URL or file name with an
extension, spaces included, while words such as "false", fragments,
templates, script URLs and spacer files are still skipped. An unmarked
image, whose src would be replaced, keeps the strict rule.

On WCXB, output is unchanged in both modes: only one Luxoft page holds
words in a marked image's attributes, and they are still skipped.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The spec for a player and its caption passed with or without the rule
that a container holding a player is content, because its "media" class
weighs nothing now and its caption keeps it. Nothing showed that the
rule mattered.

A new spec puts a player beside a download link, and two players in
their own divs. Without the rule, link density removes the first and
the empty-text check the second.

Code and output are unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A URL with a stray ")" could write a second link after the first; the
changelog's entry on disguised script links now says it no longer can.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A marked image took a lazy-loading file name only if it ended in an
extension and an optional query with no spaces. "logo.svg?text=hello
world" was dropped, though main kept it, and on Ruby 2.7 a value of
"a.svg?" repeated 16,000 times before a space took 3 s, as the regex
retried the query from every dot.

A marked image now takes any value with a dot in it, beside paths and
web URLs; words such as "false", fragments, templates and script URLs
are still skipped. The long value takes 13 ms.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A link whose text is its own URL is written as the bare URL, raw. With
data-href now read, <a data-href="mailto:[Run](javascript:alert(1))">
holding the same text was written as "mailto:[Run](javascript:alert(1))",
which Markdown reads as a script link. The same text in an href already
did this on main; data-href made it reachable from one more attribute.

A URL holding brackets, angle brackets, a backtick or a backslash is
now written as an ordinary link, whose text is escaped and whose
destination is encoded. CommonMarker reads every injection repro as
links to mailto, http or relative URLs only.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A marked image took a lazy-loading value only if it held a slash or a
dot, so "?image=7" and "image?id=7", which a loader can serve images
from, were dropped; main kept both.

A marked image now also takes a value with a query. The check for
words, fragments, templates, other schemes and spacer files is shared
by marked and unmarked images, and an unmarked image still needs an
image file or a path to replace its src.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Link text kept its brackets whenever it held as many "[" as "]", and a
backslash at its end stayed raw. "Go\" then escaped the "]" written
after it, so the link never formed and the URL was read as Markdown:
<a data-href="mailto:[Run](javascript:alert(1))">Go\</a> gave a script
link. In "]x[" the counts match but the first "]" ends the text, and an
escaped "\]" next to an unescaped "[" was miscounted.

Brackets now stay only where they pair in order, skipping escaped ones;
otherwise every unescaped bracket is escaped. A backslash left at the
end is escaped. On 20,000 random labels and mailto URLs, CommonMarker
reads one mailto link each time, with the label's own text.

On WCXB, three pages change in full mode, all for the better. Two MDN
pages wrote "[[Symbol.iterator\]()](…)", which CommonMark reads as a
stray "[" before the link; the text is now "\[Symbol.iterator\]()". An
image caption on dev/0413 gets the same fix. Main mode is unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
jasir-serpapi and others added 28 commits November 10, 2026 15:03
Each inline piece was escaped on its own, so pieces that met could
write what neither held: "Wow!" before a link wrote an image, "[1]"
before "(note)" a link, and a "<" before "javascript:alert(1)>" or
before an email address split over a <code> ("<0" + "`@h.co>`") an
autolink. A "<" then "a title='" before a link wrote an HTML tag that
swallowed the link's opening bracket and let its code write a script
link.

Where two pieces meet, a "!" before "[", a "]" before "(" and a "<"
that would open an autolink, an email autolink or a tag now get a
backslash, unless one escapes them already. A "<" before a web URL
written as text stays, as "<https://…>" is meant to link. The string is
changed only at its end, in bytes, so long text is not rescanned.

On WCXB, 2 pages change in main mode and 9 in full mode, all the same
way: "here![Learn More](…)" now reads as text and a link, not an image.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The Unreleased entry on link safety did not cover the latest fixes:
email autolinks, raw TeX, emoji alt text, trailing backslashes, long
backtick runs, adjacent code spans and text pieces side by side.

The entry now lists them.

On WCXB, output is unchanged: documentation only.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The space that keeps two code spans apart also went after a backtick
that text had escaped, which cannot join a fence: "x`<code>y</code>"
became "x\` `y`", adding a space main does not have. The rule had to
take every backtick as a fence, because code ending in a backslash
("`C:\`") looked like an escaped one.

Code that ends with a backslash is now padded like code next to a
backtick ("` C:\ `"), so a backtick escaped by a backslash is always
text, and the space goes only after a fence.

On WCXB, 3 pages change in both modes: one loses the space after an
escaped backtick, and two Windows paths ending in "\" get padded
fences, which render the same.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The check for an email address split over pieces walked back over the
characters of a local part, backticks included, so it went through the
fences of a code span and escaped a "<" inside it: "<code>&lt;</code>
user@example.com&gt;" wrote "`\<`user@example.com>", which shows the
backslash. In code it is literal, and code outranks the autolink.

The walk now pairs fences on its way back. A "<" is escaped only if no
code span closed before the address is still open there, and the walk
stops at a backtick that a backslash escapes.

On WCXB, output is unchanged in both modes.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Padding code that ends with a backslash ("` C:\ `") kept a fence apart
from an escaped backtick, but the 2 extra characters could push a
block link's text or a heading past the 300-character cutoff, which
dropped the link or turned the heading into a paragraph. Main keeps
both.

Code is written unpadded again. Inline rendering now records whether
each piece ends with a backtick that text escaped, so the space goes
after a closing fence and never after an escaped backtick, without
reading this from the string.

On WCXB, 2 pages change in both modes: the Windows paths in dev/0566
and dev/2826 lose the padding that 425ba1b added.

Main mode p50 14.5 ms, p99 122.1 ms; full mode p50 8.0 ms, p99 51.9 ms
(2008 pages).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The previous commit took a backtick as escaped only when it ended text
or an element that returned that text unchanged. Elements that copy
their text, such as <sup>, <math> without TeX or an anchor that is not
linked, lost the note, so "<sup>`</sup><code>x</code>" gained a space
("\` `x`"). The extra character could still push a block link or a
heading past the 300-character cutoff.

The string tells the two apart again: a backtick after a backslash is
text, except when it closes code that ends with a backslash. Only that
case is noted, by #code_span, and the note passes up through every
element whose text ends with that code.

On WCXB, output is unchanged in both modes.

Main mode p50 14.2 ms, p99 121.9 ms; full mode p50 7.8 ms, p99 50.0 ms
(2008 pages).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The video host pattern matched "//player.vimeo.com" anywhere in an
iframe's src, so "mailto:x//player.vimeo.com/video/[Run](javascript:…)"
was kept. An embed without a title was then written as its raw URL, as
text, and the Markdown in it parsed as a link to javascript:.

The src must now be an http(s) or scheme-relative URL whose host is the
provider's and which has a path, after the same normalization links get.
An embed without a title is written like a native player, as
"[Video](url)" with the URL as a destination, so no source is ever
written as text.

On WCXB, 6 pages change in main mode and 8 in full mode: their 8 and 16
titleless YouTube and Vimeo embeds become "[Video](…)" links. No embed
is lost or gained. WCXB F1 is unchanged (dev 0.876, test 0.900),
ScrapingHub stays 0.947, and Mozilla rises from 0.947 to 0.949 on its
embedded-video pages, where a URL scored as words.

Main mode p50 14.4 ms, p99 129.5 ms; full mode p50 8.0 ms, p99 53.8 ms
(2008 pages).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Only links without an href took their target from data-href and the
other data attributes, so "<a href=\"\" data-href=\"/article\">" stayed
unlinked.

A link whose href is empty after the normalization links get (only
spaces and control characters) now takes its target from the same
attributes.

On WCXB, output is unchanged in both modes, and so is F1.

Main mode p50 14.6 ms, p99 124.8 ms; full mode p50 8.1 ms, p99 51.7 ms
(2008 pages).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A caller converting a selection of a page passes main_content: false
and expects all of it converted. The whole-page mode still deleted
selected content: buttons and open dialogs ("Download paper", "Visible
dialog" gave ""), repeated headings ("## Size" x3 gave one), repeated
paragraphs (2 of 3), images whose 10+ character alt matched (1 of 2),
rules (a lone <hr> gave ""), and quotes of earlier text were shortened.

In full mode Cleanup now drops only punctuation-only lines: no
deduplication, repeated runs, quote shortening, rule trimming, empty
sections or repeated-image removal. The preprocessor keeps buttons (as
their text), labels, forms, textareas, outputs and open dialogs; it
still removes scripts, styles, hidden elements, inputs and closed
dialogs. Specs of main-content cleanup now render through a helper that
applies it, since their fragments are too short to pass for content.

On WCXB, main-mode output is unchanged, so F1 is too. In full mode 1707
of 2008 pages change, with 3.3% more characters: 5,492 more images
(39,796 to 45,288), 253 more rules, 499 more headings, 415 more quote
lines, plus button, label and form text.

Main mode p50 14.9 ms, p99 128.1 ms; full mode p50 7.6 ms, p99 50.3 ms
(2008 pages).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A link whose only content was an image lost its target: the image was
written before the link, and the empty link text dropped the link. In a
heading the image was dropped as well, so the title graphic and its link
both vanished.

Each image of such a link is now written as [![alt](src)](href), and a
heading made only of images keeps them as a paragraph. Image cleanup
treats a linked image as one unit, and a linked image no longer counts
as a text link that makes its own thumbnail redundant.

WCXB: 297/2008 pages change in main mode, almost all images gaining
their link. Dev F1 0.87553 -> 0.87547 (one page, whose alt "[X-Info]"
the benchmark's Markdown stripper cannot parse, -0.116; one +0.017).
WCXB test, ScrapingHub and Mozilla F1 unchanged. Main p50 14.9 -> 14.6 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A player inside a link previews the page it links to, so it was removed
whole, taking the thumbnail it holds for browsers without video, and so
the link with it.

The preview file is still skipped, but the player's own <img> and
<picture> children now stay in its place, inside the link. Its fallback
text ("Your browser does not support the video tag") is still dropped,
and only the player's own children are read, keeping nested players
linear.

WCXB: 0/2008 pages change in either mode (fallback text kept, tried
first, changed 11 main / 13 full pages with "browser does not support"
lines, so it was left out). F1 unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
With src="/placeholder-image", Nokodown took the real image from only 5
of 18 common lazy-loading attributes, and an unmarked data-src of
"portrait.svg" or "?image=portrait" lost to a thumbnail src. Amazon's
data-a-dynamic-image was ignored, and a srcset written without a space
after its commas ("650w,https://…") gave nothing.

Lazy attributes are now read in a fixed order, high-resolution names
(data-a-hires, data-src-hq) first, then the other loaders' names, and
win over any src on every image. The largest size in
data-a-dynamic-image wins, and a lazy srcset gives its largest image
when no plain attribute holds one. Srcsets are parsed as browsers parse
them. An unmarked image's src is replaced by any path, query or file
name, not only a raster file.

A value that is no image location (a word, "false", "#" or a number)
never replaces src. On WCXB that keeps 54 images from breaking (22
"false" at IKEA, "Facebook"/"LinkedIn"/"X (formerly Twitter)" social
icons, 28 numeric IDs in data-image/data-img) and loses no real image.
With mixed descriptors ("1w, 2x"), the largest number wins (40 Thomann
logos get the 2x file).

WCXB: main mode changes 2 pages (larger Amazon images); full mode 10
pages (Amazon, BigCommerce 2560w, Mises lazy srcsets, Etsy thumbnails
restored). F1 unchanged (the benchmarks strip images).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
When a page had no section with content, drop_empty_sections returned
only its H1, so "<h1>One</h1><h2>Two</h2><h3>Three</h3>" came out as
"# One".

When no block is kept and every block is a heading, the headings are the
page's outline and all of them are returned. A page with other blocks
still falls back to its H1 alone.

On WCXB, output is unchanged in both modes, and so is F1. ScrapingHub
and Mozilla are unchanged too.

Main mode p50 15.2 ms, p99 128.1 ms; full mode p50 8.5 ms, p99 54.3 ms
(2008 pages, with the table change below; noise against 15.1/124.9 and
7.9/49.6 before).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A data table with one column was always unwrapped into paragraphs, so
"<table><tr><th>Col</th></tr><tr><td>a|b</td></tr></table>" gave
"Col\n\na|b" instead of a GFM table with the pipe escaped.

A one-column table now stays a table when it has a <th> or a non-empty
caption, which name the column. Without either it is still unwrapped,
as layout.

On WCXB, 5 main-mode and 7 full-mode pages change (MDN and TC39
"Specification" boxes, ESPN standings, csrankings areas, a Go TOC); F1
is unchanged on every page, dev and test. ScrapingHub and Mozilla are
unchanged.

Main mode p50 15.2 ms, p99 128.1 ms; full mode p50 8.5 ms, p99 54.3 ms
(2008 pages).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
When the article body was chosen below a headline block inside <main>,
the H1 was left out: without a <title> the output started at the body,
and with one only the metadata title came back.

When no selected node holds an H1, the H1 of the enclosing <main> (or
role=main) opens the content, if the body follows it there. Under
title: false it stays out, as the title does.

The deck or subtitle below the headline is still left out. Keeping up
to two short plain lines after the H1 cost F1: ScrapingHub 0.947 ->
0.946 (11 pages down, worst -0.021) and Mozilla 0.949 -> 0.948 (folha
-0.018, topicseed-1 -0.016, gitlab-blog -0.013), because both
references leave the deck out of the article body. Keeping everything
between the H1 and the body cost more (byline leaks, dev/2201 -0.514).

On WCXB, 212 main-mode pages change, mostly the page's H1 replacing a
near-duplicate metadata title. Dev mean F1 0.87547 -> 0.87544, test
+0.00003; ScrapingHub unchanged, Mozilla 0.949 (v8-blog +0.002).

Main mode p50 15.8 ms, p99 138.6 ms; full mode p50 8.1 ms, p99 52.6 ms
(2008 pages; 15.2/128.1 before. The H1 lookup is one ancestor walk per
article page, so the difference is run-to-run noise.)

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A caller that keeps only the first 50,000 characters still had
Nokodown render the whole page first: 3.7 million characters for a
3.8 MB article.

max_length: nil (the default) changes nothing. An Integer n gives
exactly to_markdown(html)[0, n] and the same excerpt; other Result
fields come from metadata. A negative or non-Integer value raises.

The Renderer stops after about max(2n, 1000) characters, between two
blocks outside lists, tables, quotes, headings, links and other
elements whose Markdown depends on all their content. Cleanup keeps
only blocks the whole content's cleanup also starts with: a repeat at
the end can still begin a run, and trailing headings and rules can
still gain or lose a section. The short render is kept only when it
reaches n characters and nothing later can change its start: no image
whose copies so far are all redundant, no H1 further down that could
replace the page title, and an excerpt paragraph when metadata has
none. Otherwise the whole content is rendered once, so a page costs at
most one short render and one whole render.

Selection, not rendering, dominates main mode, so the gain there is
small; full mode gains most. tmp/bench/big.html, 20,000 paragraphs:
  main: 1072 ms -> 966 ms with max_length: 50_000
  full:  160 ms ->  71 ms with max_length: 50_000
Without max_length, every WCXB page gives the same output in both
modes, and F1 is unchanged on ScrapingHub (0.947), Mozilla (0.949),
WCXB dev (0.875) and test (0.900). With it, all 2008 WCXB pages give
the exact prefix at 0, 1, 100, 1000, 5000 and 50,000 in both modes;
at 50,000, 1 page in main mode and 2 in full need the whole render.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Scorer#within_root? listed every ancestor through Node#ancestors, which
builds a NodeSet of all of them, for each scored paragraph and again
for each ancestor while climbing. It now walks up the parents and
stops at the root. Readability's comma count also uses String#count
instead of splitting the text: the text is never empty there, so the
pieces between commas are always one more than the commas. A spec
checks the count against Readability's rule, trailing comma included,
and one checks that scoring no longer lists ancestors per paragraph.

On all 2,008 WCXB pages no output changes, in either mode, and F1 is
unchanged on ScrapingHub (0.947), Mozilla (0.949), WCXB dev (0.875)
and test (0.900).

Ruby 2.7.8, median of 5: tmp/bench/big.html main mode 1195 -> 1141 ms.
Wikipedia Coffee main mode 276 -> 273 ms (median of 10, within noise).
WCXB main mode p50 15.3 -> 15.2 ms, p99 129.1 -> 128.3 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Boilerplate matched each class name of a page against the chrome and
noise patterns, about 4 µs a name that matches none: 20 ms on
Wikipedia's Coffee article, whose 5,513 names nearly all miss. The
secondary-content pattern ran over every signature on four passes.

Every name these patterns match holds one of a few dozen words ("nav",
"footer", "cookie", "share", "comment", ...). Patterns.furniture? and
Patterns.secondary? now look for those words first, through a regex
with shared prefixes, and run the full pattern only on a hit; on the
Coffee article 367 of 5,513 names do. A spec spells out every name the
patterns can match and checks that each holds a word, so a pattern
that gains a name without its word fails. Readability's NEGATIVE
weight matched "hid" as a whole class name through four anchored
branches; one branch with lookarounds matches the same values in half
the time.

On all 2,008 WCXB pages no output changes, in either mode, and F1 is
unchanged on ScrapingHub (0.947), Mozilla (0.949), WCXB dev (0.875)
and test (0.900). Over the 319,056 signatures and 334,980 names of
WCXB, the prefiltered checks give the same answer as the patterns.

Ruby 2.7.8: Wikipedia Coffee main mode 268 -> 244 ms (median of 10),
books.toscrape 2.2 -> 2.1 ms. WCXB main mode p50 15.2 -> 14.0 ms, p99
128.3 -> 117.8 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Key assignment, heading anchors, card runs, the scorer's comment
regions, bylines and share widgets each listed every element through
css("*"), "[class]" or "[id]". Lexbor's selector engine took about
8.5 ms per pass on Wikipedia's Coffee article; a walk from Ruby over
first and next element children takes under 2. Dom.elements gives the
elements in the same document order, and the callers that selected by
attribute now check it with key?. Stats walks children through
next_sibling instead of building a NodeSet for each element.

Specs check Dom.elements against css("*"), including the copies Lexbor
makes of a misnested formatting element, and that keys follow document
order. Over 1.39 million elements of every 4th WCXB page and their
first 20 sections, the walk and css("*") give the same order.

On all 2,008 WCXB pages no output changes, in either mode, and F1 is
unchanged on ScrapingHub (0.947), Mozilla (0.949), WCXB dev (0.875)
and test (0.900).

Ruby 2.7.8: Wikipedia Coffee main mode 248 -> 215 ms, full mode 98 ->
89 ms (median of 10); books.toscrape 2.1 ms. WCXB main mode p50 14.0
-> 13.5 ms, p99 117.8 -> 112.6 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Node#traverse builds a NodeSet of every node's children, text nodes
included, and recurses through NodeSet#each. Scoring paragraphs, finding
the paragraph keys of a root with misnested formatting copies, and the
best score within a sibling only need the elements, children before
their parent. Dom.elements_bottom_up walks first_element_child and
next_element with a stack and yields them in the same order.

On the 2008 WCXB pages the Markdown is identical in both modes, and F1
is unchanged on every corpus. Scoring the paragraphs of the 5.4 MB
article takes about 75 ms instead of 92 ms.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The Unreleased section did not mention max_length, blank hrefs, links
made only of images, thumbnails of players inside links, iframe embeds,
heading-only pages, one-column tables or the headline above an article
in main. The speed note now gives the numbers at this commit: Wikipedia
Coffee 364 to 214 ms in main mode, WCXB 30.2 to 22.0 ms per page (dev).

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A player inside a link was replaced by its thumbnail before hidden
elements were removed, so the thumbnail of a player marked hidden,
aria-hidden, display:none or with a hiding class lost those attributes
and was rendered. Such a player now goes with its thumbnail.

WCXB: no page changes in either mode; quality unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A marked image's lazy srcset and the keys of data-a-dynamic-image
replaced src without the checks the plain lazy-loading attributes get,
so "false 2x" or {"123": [600, 600]} turned a real src into "/false"
or "/123". A lazy srcset is now judged as the plain attributes are, and
a dynamic image map gives only an image location.

WCXB: no page changes in either mode; quality unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
…ails

data-thumb came before data-hi-res-src and data-large-src, and the
largest size in data-a-dynamic-image won over every attribute, so an
image with both a thumbnail and a high-resolution URL got the thumbnail.
The high-resolution names (data-a-hires, data-src-hq, data-hi-res-src,
data-large-src, data-original and the like) are now read first, then
data-a-dynamic-image, then the other loaders' names, with data-thumb
last among those an unmarked image reads.

WCXB: no page changes in either mode; quality unchanged. Of 3,687 lazy
images, the same 3,588 pick the same source as before.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
A pipe after a backslash was left as it was, on the assumption that the
backslash already escaped it. But a GFM table reads every "\|" as an
escaped pipe before the cell's inline Markdown, so the backslash of
"a\|b" was lost: the cell read "a|b". Every pipe now gets its own
backslash. Single-column tables made this reachable from code that was
written as a code span before.

WCXB: main mode unchanged; in full mode one page (a Wikipedia navbox of
nested tables) writes "\\|" for "\|" and renders the same HTML.
Quality unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
The traversal specs claimed to cover copies of a misnested formatting
element, but their link legally holds a block, so Lexbor made no copies.
They now use a link cut by a closing block, and check that its copies
share a key before comparing the walk with css("*") and traverse.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
…gelog

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
Since every pipe in a cell is escaped, a table nested in a cell was
escaped twice: once in its own cells and again as text in the outer
cell. A GFM table removes only one table escape, so code in the inner
table gained a backslash ("a|b" read "a\|b"), and one more per level of
nesting. Only the outermost table escapes pipes now; a nested table's
cells are written as they are, for the outer cell to escape.

WCXB: main mode unchanged; in full mode the one page the previous
commit changed (a Wikipedia navbox of nested tables) is back to its
earlier output. Quality unchanged.

Co-authored-by: Codex + Claude Opus 5.5 <codex@openai.com>
@jasir-serpapi
jasir-serpapi merged commit 3dc2249 into main Nov 12, 2026
4 checks passed
@jasir-serpapi
jasir-serpapi deleted the jasir/max-length-and-fixes branch November 12, 2026 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant